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Several years igo, California^ along with otb^j 
states, ndisccvered thar rbere vas a layering effect of categoricap^ 




categoi 

efforts c^ea^ed probleits of sufficient aagnitade to variant action zp 
relieve school districts. Throngh ref ora efforts sdrools xere 
required to protide- thorough needs assessments of tiieir pnpils to 
develjop plans in vhich tke various .funding sources could be brought 
together into a <»herent/vhole to^aeet khe established needs of the - 
student^. Fhile this refora aoveaent has forged al^ead the problea 
r-eaains that there are still unique evaluation reguireaents for each 
.of t?e individual prograas. The problem arises of atteaptii^g to aake 




presents 

kind of infor-aaticJn policy makers need is not of the descriptive 
nature^ vhiqi; -has -topically chara^erized an evaljaatibn^ tut father 
one <rhich can yiela inferences abdut contrasts between prograas. A 
c<5vei&ent toward a ^'hyper^e valuation" 'whiph is aore akin to- 
etperiaental or research design is foreseen. (2C) 
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Tne'sz^ title of this p^er ai^t logically be, ^'Hard Tiises l^ad 
to Sard Decisions.-"' You are all faniliar vith t±ie evolution *in 
the past decade of the evaltiaticn and acco'antabiIi|rf movement 
which vas generated primarily by passage of the eler^entary and 
secondary e5iucatioB prograiES. In" addition to these ongoing 
requirepehts of reporting .to the federal govems^ent. In Calif or- 
p.ia ve have had certain developisents «rhich have aisplified the 
problea. ' . ' 

Ihese developments can be called reforni efforts. Several yeS^ 
ago, we; along. vith other states, discovered that there was 3. 
layering effect of categorical prograsis-, both state and federal. 
Tae majority of these prograsis were ainied at unique topulations , 
SToch' as the* disadvantaged, 3ilingual--stiident^ with unique edu- 
c^^i^i^l needs . Ihis proliferation and ^e evaluation require- 
isents inherent in each of the categorical efforts created 'proi>^ 
lens of sxifficient aagnitxaxie to warrant action to relieve school 
districts. * ' " ' — ^ ' 

• Aaong these problesis were the proliferation of paper work and 
the need to assess nrultiple programs that $ere dealing with the 
sake pupil populations. An extr^e exai^le.is that in a single 
second grade classroom in one niajor jaetropjcxlitan area, diere 
were seven separately fimded prograias, eaoh with unique applica- 
tion, program development aiiii* evaliiation |equirements . ike 
absurdity of such a situation is' self-evident. Three years ago 
a group of prominent educators conceptualjzed the idea of a 
liiassive refers,, beginning in the elemental grades. Ihis effort 
has- -been Jinown as the Early Cnildhood ^dupation Reform (ECE>. I 
purposely tise the word 'reform'^ rather than program becaiise ECE 
attempts to make substantial changes in the total education 
program by not only instituting, changes ifi instructional practices 
but also addressing the problem of ^ftag^neiited efforts to assist 
student;s. Under this reform concept schools are required to pro- 
vide thorough needs assessments of their oupil populations to 
develop plans in which the vkrious funding sources can be brought 
together dnto- a coherent whole to meet th^ established needs of 
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*A presentation in a. symposium, ^ End of Affluence : Educational 
Evaltiation in -Tl;^ Koney Tiroes , Annua l"~Hee ting of the~^erican 
Educational Kesdarih Association, San Francisco, California. 
April 19-23, 1976/ 



rnejr stTaears, " "Spjs, «itaia EGE ve see tSs ecaiinaricn or* - 
i^tle I. S£lia^l,edi:^riim^ bqt± ,j-ederal state, oyr ovn^ 
iuiiicatxciialiy Sisaj2vantag^Toiit3K|>rograis. special ^Pjxfiny 
?rogr^,_ari^ certain optiers broii^f toglther in a.miified 
. erfort. Maxle tiiis ^^jfei^t^ - forged asead/ and gained - 



^i£t ourselves cast into the ^tistion of att4E33tia£ to taake 
asGnaDle ind^Eents' about the -ECtal^ effectiveness' of a reforta 



We 

reascsnabL 

exxoru and jet ha-rxng to ireet the • legislati-zelj established 
requiresLencs or unique f-aiding sb-orce evaliiations . 

Tnis effort rapidlj becoites nonsensical vhen vou consider rvo 
e-:;i5^nt racts- fl) the -funding soiirces do nor define as such a ' 
unique instructional program They a-e merely ^^ehicles bv vhich - 
soiiars are transrerred from one treasury 'to another and' -(2) the 
populations for ^iica 'the prograsas are designed are not uniquely 
einerent m the various schools in which the orograins are iisole- 
EKited. Beca'-ise of -the hard money times, we are asked now to* 
'maKe lUterprogrsa cot^jari sons- -that is, to contrast effects o^ * 
one categorical program vith those- of another, vhen in fact they 
are i23)acting the same population^ and most orobablj using the 
same instructional interventions. Demands are "made on us by the 
legislature and regulatory and control agencies "in the name or 
sound public policy, to determine the relative' worth of the com- 
paraole programs, is the Hilier-Unruh Reading Program more 
errective or less effective than a reading program funded by the 
state pro-am for disadvantaged youth? ' ^ ^ ^ «u d> tne 

Under the historical mode ^f federally funded categof ical programs', 
dollars were apportioned on a formulaic basis. Stey were- entrtie- 
ment programs. Districts were entitled to receive money 
and it still \T.rtually takes,a felonious acton the part'of a 
district not to get these dolla#. No judgment of relative success 
or failure is really necessary for these -programs -to be contiriued 
The state cate^oricai programs however, are based on a contrary 
vtBw^That IS, these, programs must demonstrate their effectiveness 
11 they are .to oe continued. .Thus, in addition to tfie between or 
mcerprogram comparisons^ we now must make some jwdgment ^bout the 
relative effectiveness .gf schools.. Such a process is obviously 
. rrougnt with peril. Policy makers assume-that we are -able -to ma?^ 
a true evaluation- -a complet>eiy accurate ranking of effectiveness 
irom the greatest to the leastr-regardless of the pupil oopulation, 
the instruments involved,^ and all the intervening variables. 

The situation is not unlike that which* occurs with regard to inter- 
pretation of grade equivalent scor'es, when people know that John's 
grade equivalent of 5.6 is obviously superior to Jane's equivalent 
or . ^ 




Tnis. sasimder standing leads to thel^aat' ox sy litany of problems 
of appropriate ips tnrr^n tat ion ^stTanalysis for iiS€^ in evaluating 
and reporting results. Historically, ve have reqtiired a pre-po^t 
procednre using nons- referenced tests. We are able vo .tak^ these 
various^ test results and put them on a cosnaoq metric stich as a. 
standarjd score/ yhici\ I believe is infinitely iDore sensible than " 
other scales. Eoveyer, the problem '?:>f'' test content, the technical 
-^i^lre ^rs of nomo , and a syriad of others resarn to be contended \ 
withr^ 4vhat have we done to atteirot to resolve these dilensaas? 
The first jnove was tp go to a consolidated application evaluation 
'and applicant agencies could in one docuiaent apply for ail categdr- 
icai funds. Ve 's^y&Ci to, a consolidated evaluation where .in 6ne 
docukest districts could* )ifK^ or t to us data necessary for analysis, 
ne issijed guidelines to districts where one assessHient could 
used for a variety. of qat4fe^orickr prograiES. We are conteirolating 
and weighing rhe iserits of using our state assesstsent prograsj as 
the prinie vehicle to collect coisnon achievement inronsation in thfe 
eleHienpary grades. mese provide partial solutions to the problem 
but the tough •qt>^st ions have yet to be answered. 

We seek the advice of vafioizs groiips to help us in ^ the probJLea of 
analysis and the presentation of .appropriate infonnatfon to the 
body politic. I use the phrase '*body politic" in the-"b^oadest 
-sense, since it ±s increasingly clear t±at we are becoming cast 
into a political i^de. Every evaluation* report- beooses a political 
docuiEent, aind its worth is therished by scnrk and ridiculed* by 
others. The devaluation reports ^re not used as the sole input by 
which broad policy dteci^ions arfe'Wd?. Evaluation repor?:s are not 
>asBd upon ri^roiis experisental ^dsi^xsrjet inferential state- 
ments are demonstrated. These self-evident facts are difficult to 
comsijmie^ coizsmiity spends an inordinate aaount 

* of time"^j^ai4rfyihg and issuing caveats about eval\iati^n reporting. 
The net •effect is to reduce the credi-Bility of evaluaticai reports. 

/- 

Somehow the credibility* of evaliialfion must be re-established. 
Academic nit-^picking as to the relative Superiority of analytical 
methods does not ass^ist any agency in attempting to coismunicate 
infqrmation to* policy makers. * . ^ 

- Evaluators cannot agree on a desi^,'^e appropriate- instrumenta- 
tion, or the appropriate analytical procedure. Phillips, in his 
paper, *Vhen Evaluators Disagree," states the problem well. Hrs 
solution is to establish panels of expert3 with common philosophical 
backjgrounds . I'm not sure we can find five people who meet this i 
criterion. ^ • - * ^ \ 

,The c6^l^)et^tion fpr money will increase. Evaluations will be 
>a^ke<J tTo pL^y an increasingly potent role in decision -making. 

This dileimna pr^sent^.it^elf : The kind of information policy makers 
need-^^r feel they needis^ nbt^of this descriptive nature, which 
has typically characterized an evaluation^, but rather one which 
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can yield inferences about and contrasts between orosraias I 
see a-aovgmenr Qlearly underwav towards a *li#per-^.Tal^^tion" 
i'w^".^''^''^'''^ '^^ classical evaluation design and is aore 
atcm to an experiirental or r^earcn desi^. - 

of effort wili>ield usable results, 
-ef-fi^s "^^^ ^ ^^y^^^ "^^^^ present 
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